Method and apparatus for detection of false alarm obstacle

ABSTRACT

The embodiment of the present application provides a method and apparatus for detection of false alarm obstacle, and relates to the field of identification and detection technology, in order to improve accuracy for the obstacle detection. The method includes: obtaining a first view at a moment t and a second view at a moment t- 1,  collected by a camera; determining location information of the same obstacle to be detected in the first view and the second view; determining motion information of the camera from moment t- 1  to the moment t; judging whether the location information of the same obstacle to be detected in two views being matched with the motion information of the camera; and if not, judging the obstacle to be detected as a false alarm obstacle. The embodiment of the present application is applied to obstacle detection.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application under 35 U.S.C.§ 120 ofPCT application No. PCT/CN2016/112632 filed on Dec. 28, 2016, thecontents of which are incorporated herein by reference.

FIELD OF THE INVENTION

The present application relates to the field of identification anddetection technology, and in particular, to a method and apparatus fordetection of false alarm obstacle.

BACKGROUND OF THE INVENTION

At present, obstacle detection is one of the most important parts in anytechnical field of unmanned driving, aided driving, drone navigation,blind guiding, and intelligent robots. For example, intelligent robotsor unmanned vehicles and the like perform autonomous navigation inunknown environments to sense the surrounding environments, and a systemis needed to provide information such as obstacles and roads in theenvironments. In recent years, as the rapid development of the computerimage processing technology, the application of visual sensors inobstacle detection has been paid more and more attention. Therefore, atthe current, the mainstream method for obstacle detection is a methodfor obstacle detection based on visual sensors. However, due to theinfluence of noise, errors and the like, the obstacle detection resultis not accurate enough.

SUMMARY OF THE INVENTION

The embodiment of the present application provides a method andapparatus for detection of false alarm obstacle, in order to improve theaccuracy of obstacle detection.

To achieve the above objective, the embodiment of the presentapplication adopts the following technical solutions:

In a first aspect, a method for detection of false alarm obstacle isprovided, including:

obtaining a first view at a moment t and a second view at a moment t-1,collected by a camera;

determining location information of the same obstacle to be detected inthe first view and the second view;

determining motion information of the camera from moment t-1 to themoment t;

judging whether the location information of the same obstacle to bedetected in two views being matched with the motion information of thecamera; and

if not, judging the obstacle to be detected as a false alarm obstacle.

In a second aspect, an apparatus for detection of false alarm obstacleis provided, including:

an obtaining module, configured to obtain a first view at a moment t anda second view at a moment t-1, collected by a camera;

a determining module, configured to determine location information ofthe same obstacle to be detected in the first view and the second viewobtained by the obtaining module;

the determining module is further configured to determine motioninformation of the camera from moment t-1 to the moment t; and

-   -   a judging module, configured to judge whether the location        information of the same obstacle to be detected in two views        obtained by the determining module being matched with the motion        information of the camera, and if not, judge the obstacle to be        detected as a false alarm obstacle.

In a third aspect, an electronic device is provided, and the structureof the electronic device includes a processor, configured to support theapparatus to execute corresponding functions in the above method. Theapparatus may also include a memory, which is coupled with theprocessor, storing computer software codes used by the above apparatusfor detection of false alarm obstacle, and including a program designedto execute the above method.

In a fourth aspect, a computer storage medium is provided, for storing acomputer software instruction used by an apparatus for detection offalse alarm obstacle and containing program codes designed to executethe above method.

In a fifth aspect, a computer program is provided, which is capable ofbeing directly loaded in an internal memory of a computer and containingsoftware codes, and the computer program may implement the above methodafter being loaded and executed by the computer.

In a sixth aspect, a robot is provided, including the electronic deviceaccording to the third aspect.

In the prior art, due to the influence of noise, errors and the like,false factors, that is, false obstacles, exist in the obstacles detectedin an obstacle detection process. In order to detect the falseobstacles, according to the solution provided by the presentapplication, the first view at the moment t and the second view at themoment t-1, collected by the camera, are obtained, then the locationinformation of the same obstacle to be detected in the first view andthe second view is determined, the motion information of the camera frommoment t-1 to the moment t is also determined, then whether the locationinformation of the same obstacle to be detected in two views beingmatched with the motion information of the camera is judged so as tojudge whether the obstacle to be detected is the false alarm obstacle,and if the location information of the obstacle to be detected in thetwo views being not matched with the motion information of the camera,the obstacle to be detected is judged as the false alarm obstacle,therefore the false alarm obstacle may be eliminated, and the accuracyof obstacle detection is ensured.

BRIEF DESCRIPTION OF THE DRAWINGS

To illustrate technical solutions in the embodiments of the presentapplication more clearly, a brief introduction on the drawings which areneeded in the description of the embodiments or the prior art is givenbelow. Apparently, the drawings in the description below are merely someof the embodiments of the present application, based on which otherdrawings may be obtained by those of ordinary skill in the art withoutany creative effort.

FIG. 1A is a corresponding relationship diagram of a disparity and adepth of the same object shot by a left camera and a right camera in abinocular camera provided by an embodiment of the present application;

FIG. 1B is a top view of FIG. 1A;

FIG. 2 is a schematic diagram of a method flow of a method for detectionof false alarm obstacle provided by an embodiment of the presentapplication;

FIG. 3 is a schematic diagram of a matching flow of motion informationprovided by an embodiment of the present application;

FIG. 4 is a schematic diagram of another matching flow of motioninformation provided by an embodiment of the present application;

FIG. 5A is a left view obtained by the left camera when the same sceneis shot by both of the left camera and the right camera in the binocularcamera provided by the embodiment of the present application;

FIG. 5B is a right view obtained by the right camera when the same sceneis shot by both of the left camera and the right camera in the binocularcamera provided by the embodiment of the present application;

FIG. 6 is a disparity map corresponding to left and right views shown inFIG. 5;

FIG. 7 is a schematic diagram of a method flow of an obstacle detectionmethod provided by an embodiment of the present application;

FIG. 8A is a first sub-disparity map corresponding to the disparity mapshown in FIG. 6;

FIG. 8B is a second sub-disparity map corresponding to the disparity mapshown in FIG. 6;

FIG. 8C is a third sub-disparity map corresponding to the disparity mapshown in FIG. 6;

FIG. 8D is a fourth sub-disparity map corresponding to the disparity mapshown in FIG. 6;

FIG. 9 is a schematic diagram of areas where obstacles are located inthe disparity map shown in FIG. 6;

FIG. 10 is a structural schematic diagram of an apparatus for detectionof false alarm obstacle provided by an embodiment of the presentapplication; and

FIG. 11 is a structural schematic diagram of an electronic deviceprovided by an embodiment of the present application.

DETAILED DESCRIPTION OF THE INVENTION

Some of terms involved in the present application are explained below tofacilitate the understanding of readers:

The camera in the present application may be a normal monocular cameraand may also be a binocular camera. The binocular camera is a cameraformed by combining two cameras with the same parameters at a certaininterval. Generally, the left camera and the right camera in thebinocular camera are usually disposed on the same horizontal line, sothat the optical axis of the left camera is parallel to that of theright camera. Accordingly, the binocular camera may simulate an angledifference caused by human eyes so as to achieve a stereoscopic imagingor field depth detection effect.

“A disparity value” refers to a difference value between two horizontalordinates for the same pixel point in the two images obtained when thesame target is shot by both of the left camera and the right camera inthe binocular camera, that is, the disparity value of the pixel point,and correspondingly, the disparity values of all pixel points in the twoimages form a disparity map.

It can be seen from the corresponding relationship diagram of thedisparity and the depth shown in FIG. 1A and FIG. 1B, if O_(L)represents the location of the left camera, O_(R) represents thelocation of the right camera, f is used for representing a focal lengthof the lens between the left camera and the right camera, and Brepresents a baseline distance, which is equal to the distance betweenconnecting lines of projection centers of the left camera and the rightcamera, specifically:

It is assumed that the left and right cameras view the same featurepoint P(x_(c), y_(c), z_(c))of a space object at the same moment. z_(c)may generally be regarded as the depth of the feature point and is usedfor representing the distance between the feature point and a planewhere the left and right cameras are located. The images of the featurepoint P are respectively obtained on a “left eye” and a “right eye”,that is, the projection points of the feature point P on the left andright cameras are P_(L)(x_(L), y_(L)) and P_(R)(x_(R),y_(R)). If theimages of the left and right cameras are on the same plane, then a Ycoordinate of the feature point P is the same as the Y coordinates ofimage coordinates P_(L) and P_(R), it may be obtained according to atriangular geometric relationship that:

$\begin{matrix}{{x_{L} = {f\frac{x_{c}}{z_{c}}}};} & \left( {{Formula}\mspace{14mu} 1} \right) \\{{x_{R} = {f\frac{\left( {x_{c} - B} \right)}{z_{c}}}};} & \left( {{Formula}\mspace{14mu} 2} \right) \\{y_{L} = {y_{R} = {f\frac{y_{c}}{z_{c}}}}} & \left( {{Formula}\mspace{14mu} 3} \right)\end{matrix}$

Since disparity Disparity=x_(L)−x_(R), three-dimensional coordinates ofthe feature point P in a camera coordinate system may be calculated as:

$\begin{matrix}{{x_{c} = \frac{B \times X_{L}}{Disparity}};} & \left( {{Formula}\mspace{14mu} 4} \right) \\{{y_{c} = \frac{B \times y_{L}}{Disparity}};} & \left( {{Formula}\mspace{14mu} 5} \right) \\{{z_{c} = \frac{B \times f}{Disparity}};} & \left( {{Formula}\mspace{14mu} 6} \right)\end{matrix}$

Therefore, based on the above formulas, since the baseline distance Band the focal length F are determined for the binocular camera, that is,the disparity value and the depth are in an inverse relationship, sothat the depth value of the pixel point may be determined by thedisparity value.

It should be noted that, since the binocular camera in the presentapplication simulates the human eyes to collect images, the left cameraand the right camera in the binocular camera in the present applicationare disposed on the same horizontal line. The optical axes of which areparallel, and there is a certain distance between them, therefore, thedisparity mentioned in the present application mainly refers tohorizontal disparity.

The term “and/or” in the present context is merely an associationrelationship describing associated objects and indicating the presenceof three relationships, for example, A and/or B, which may indicate thefollowing three conditions: A exists separately, A and B exist at thesame time, and B exists separately. In addition, the character “/” inthe present context generally indicates an “or” relationship betweenfront and back associated objects. Unless otherwise stated, “a pluralityof” in the present context refers to two or more.

It should be noted that, in the embodiment of the present application,the words “exemplary” or “for example” or the like are used for meaningexamples, example illustration or illustration. Any embodiment or designsolution described as “exemplary” or “for example” in the embodiment ofthe present application should not be construed as be more preferred oradvantageous than other embodiments or design solutions. Properlyspeaking, the words “exemplary” or “for example” or the like areintended to present related concepts in a specific manner.

It should be noted that, in the embodiment of the present application,the meaning of “a plurality of” refers to two or more unless otherwisestated.

It should be noted that, in the embodiment of the present application,“of (English: of)”, “corresponding (English: corresponding, relevant)”and “corresponding (English: corresponding)” may sometimes be mixed foruse. It should be noted that, when the difference is not emphasized, themeanings to be expressed for the above terms are the same.

The technical solutions in the embodiments of the present applicationare described below, in combination with the drawings in the embodimentsof the present application. Apparently, the embodiments described beloware merely a part, but not all, of the embodiments of the presentapplication. It should be noted that some or all of the technicalfeatures in a plurality of arbitrary technical solutions provided belowmay be combined to form new technical solutions as long as no conflictis generated.

The executive body of the method for detection of false alarm obstacleprovided by the embodiment of the present application may be anapparatus for detection of false alarm obstacle or an electronic devicethat may be used for executing the above method for detection of falsealarm obstacle. The apparatus for detection of false alarm obstacle maybe a central processing unit (Central Processing Unit, CPU), acombination of the CPU and a memory and other hardware, or may be othercontrol units or modules in the above terminal device.

Exemplarily, the above electronic device may be a personal computer(personal computer, PC), a netbook, or a personal digital assistant(English: Personal Digital Assistant, abbreviation: PAD), a server orthe like, which may analyze left and right views collected by thebinocular camera by using the method provided by the embodiment of thepresent application, or the above electronic device may be a PC, aserver or the like that is installed with a software client or asoftware system or a software application capable of processing the leftand right views collected by the binocular camera by using the methodprovided by the embodiment of the present application, a specifichardware implementation environment may be a general computer form, oran ASIC form, or an FPGA, or some programmable extension platforms suchas a Xtensa platform of Tensilica and the like. For example, the aboveelectronic device may be integrated in an unmanned aerial vehicle, ablind navigator, an unmanned vehicle, a smart phone and other devices orinstruments needing to detect obstacles.

Based on the above contents, the embodiment of the present applicationprovides a method for detection of false alarm obstacle, and as shown inFIG. 2, the method includes the following steps:

S101. obtaining a first view at a moment t and a second view at a momentt-1, collected by a camera.

The moment t is a current moment, and the moment t-1 is a previousmoment of the current moment.

The first view and the second view in the present application are imagescollected by the camera in a predetermined scene at the moment t and themoment t-1. Exemplarily, the first view and the second view in thepresent application may be 2D images, that is, may be views collected byan ordinary camera at the moment t and the moment t-1, and may also be aleft view or a right view collected by a binocular camera at the momentt and the moment t-1; or, the first view and the second view in thepresent application may also be 3D images, that is, may be a disparitymap corresponding to the left view or the right view collected by thebinocular camera at the moment t and a disparity map corresponding tothe left view or the right view collected by the binocular camera at themoment t-1, which is not limited herein.

S102. determining location information of the same obstacle to bedetected in the first view and the second view.

S103. determining motion information of the camera from moment t-1 tothe moment t.

The motion information of the camera in the present application may bemotion state information of the camera from the moment t-1 to the momentt, and may also be the motion information of an object in the first viewand the second view. Exemplarily, the motion information described aboveincludes an affine matrix and a displacement matrix between adjacentframes or two frames of images spaced apart by a predetermined interval.

S104. judging whether the location information of the same obstacle tobe detected in two views being matched with the motion information ofthe camera, and if not, judging the obstacle to be detected as a falsealarm obstacle.

Optionally, on the processing of the step S104 in the presentapplication, that is, the process of judging whether the locationinformation of the same obstacle to be detected in two views beingmatched with the motion information of the camera may be implemented inthe following two implementation:

First implementation:

In the embodiment, the motion information of the obstacle to be detectedmay be directly matched with the motion information of the camera todetermine whether the two meet the same motion rule, so as to judgewhether the location information of the obstacle to be detected in thetwo views being matched with the motion information of the camera.

Exemplarily, as shown in FIG. 3, the step S104 specifically includes thefollowing steps:

S104 a 1. determining the motion information of the obstacle to bedetected according to the location information of the obstacle to bedetected in the two views.

S104 a 2. judging whether the motion information of the obstacle to bedetected is consistent with the motion information of the camera.

S104 a 3. when the motion information of the obstacle to be detected isinconsistent with the motion information of the camera, judging that thelocation information of the obstacle to be detected in the two viewsbeing not matched with the motion information of the camera.

Second implementation:

In the present implementation, the estimated location information of theobstacle to be detected at moment t may be estimated based on the motioninformation of the camera, thereby judging whether the estimatedlocation information is consistent with the actual location informationof the obstacle to be detected at the moment t, so as to judge whetherthe location information of the obstacle to be detected in the two viewsbeing matched with the motion information of the camera.

Exemplarily, as shown in FIG. 4, the step S104 specifically includes thefollowing steps:

S104 b 1, estimating estimated location information of the obstacle tobe detected in the first view according to the motion information of thecamera and the location information of the obstacle to be detected inthe second view.

S104 b 2. judging whether the estimated location information of theobstacle to be detected in the first view is consistent with thelocation information of the obstacle to be detected in the first view.

S104 b 3. if the estimated location information of the obstacle to bedetected in the first view is inconsistent with the location informationof the obstacle to be detected in the first view, judging that thelocation information of the obstacle to be detected in the two viewsbeing not matched with the motion information of the camera.

Exemplarily, if the difference between an estimated location of theobstacle to be detected in the first view and the location of theobstacle in the first view is larger, a detection error in an area wherethe obstacle to be detected is located may be determined, the obstacleis a false alarm obstacle, and the location information of the obstacleto be detected is removed from a location information list of an objectin the first view.

For example, at the moment t-1, a certain contour area on the secondview is considered as the area of the obstacle to be detected, and acoordinate P1(x, y) of the contour area of the obstacle to be detectedis determined, wherein (x, y) represents the coordinate values of a leftcorner in the top area of the contour; then, P1(x, y) is projected ontoa corresponding location on the image at the moment t according to themotion information of the camera and is marked as P1′(x, y); acoordinate P2(x, y) of the contour area on the first view is determined;and finally, a difference value between the two coordinates P1′(x, y)and P2(x, y) is calculated so as to obtain an absolute difference valueT of the two coordinates.

T=|P′ ₁(x,y)−P ₂(x,y)|  (Formula 7)

In addition, we also set a judgment threshold value Tre, and when thecalculated threshold value T>Tre, it is judged that the area of theobstacle to be detected is a false alarm area and is deleted from acontour list. Otherwise, it is judged that the area is an obstacle areaon the second view. By this way, the false alarms in obstacle detectionare greatly reduced.

So far, all the obstacle information corresponding to the image at themoment t are subjected to a false alarm elimination operation, and thefalse obstacle information are removed.

It can be seen from the above contents that, the solution may alsoperform motion estimation on adjacent frames or two frames of viewimages spaced apart by the predetermined interval, therefore the falsealarm obstacles appearing in the obstacle detection may be eliminatedbased on the motion information, thereby improving the obstacledetection accuracy.

Optionally, if the camera is a binocular camera, then the first view inthe step S101 is a disparity map of the left view and the right viewcollected by the binocular camera at the moment t, and the second viewis a disparity map of the left view and the right view collected by thebinocular camera at the moment t-1. For example, FIG. 5a shows a view aas a result of a collection of a scene A by the left camera of thebinocular camera at the moment t, and FIG. 5b shows a view b as a resultof a collection of the scene A by the right camera of the binocularcamera at the moment t, and a stereo matching is performed on the view aand the view b to obtain the corresponding disparity map c, and whichmay be specifically referred in FIG. 6.

Based on this, the step S103 specifically includes the following steps:S103 a. determining the motion information of the camera according tothe location information of other objects except for the obstacle to bedetected in the first view and the location information of the otherobjects in the second view.

Exemplarily, when the motion information of the camera is determined, itmay be obtained by the following two examples:

First implementation:

In the present embodiment, the motion information of the camera isestimated, by the motion information of the object shot by the camerabeing mainly tracked and using a common 2D image motion target trackingmethod (such as an optical flow method, TLD and the like). It should benoted that, in the common 2D image motion target tracking method, themotion information of the object is generally estimated based on thelocation information of the object on a 2D image, while the locationinformation of the object on the 2D image usually needs to be determinedaccording to the location information of a corresponding 3D image (e.g.,a disparity map) thereof.

Exemplarily, the step S103 a specifically includes the following steps:

A1. determining the location information of the other objects in a thirdview and the location information of the other objects in a fourth view,according to the location information of the other objects except forthe obstacle to be detected in the first view and the locationinformation of the other objects in the second view.

Wherein, the above third view is the left view corresponding to thefirst view, and the fourth view is the left view corresponding to thesecond view; or, the third view is the right view corresponding to thefirst view, and the fourth view is the right view corresponding to thesecond view.

A2: performing motion estimation on the other objects in the third viewand the fourth view according to the location information of the otherobjects in the third view and the location information of the otherobjects in the fourth view, so as to obtain motion information of thecamera.

In general, an estimation of 2D image motion may be roughly divided intotwo parts: feature point detection and feature point tracking. Atpresent, feature point extraction algorithms which are more successfulinclude: Harris corner detection algorithm, SUSAN algorithm and SIFTalgorithm or the like. Feature point tracking matching algorithmsinclude: KLT (Kannade-Lucas-Tomasi) algorithm. It should be noted that,since the above feature point extraction algorithms and the featurepoint tracking matching algorithms are relatively evolved feature pointextraction algorithms. References may be incorporated in the descriptionfor the existing feature point extraction algorithms, and therefore,details related to are not described herein again.

After the feature points in a frame of image are extracted, trackingmatching must be performed in subsequent images to obtain potentialcorresponding points.

if the KLT algorithm is taken as an example; and

it is assumed that the locations of two consecutive frames of imagesI(x, y) and J(x, y) satisfy:

j(Ax+d)=1(x)   (Formula 8)

It is assumed that the

$A = \begin{bmatrix}d_{11} & d_{12} \\d_{21} & d_{22}\end{bmatrix}$

represents a 2×2 radiation transformation matrix, d=[d₁d₂] represents asa translation matrix. In addition, It is assumed that a feature windowof I(x, y) is A(x) and the feature window of J(x, y) is B(x), then:

α=∫∫_(w) [B(Ax+d)−A(x)]² w(x)   (Formula 9)

When the Formula 9 takes the minimum value, an affine matrix A and adisplacement matrix d may be derived in a derivation manner.

Second implementation:

By means of the motion information of the camera in the presentembodiment, the motion state information of the camera at the moment tand the moment t-1 may be directly calculated based on a vSLAM or IMUmethod.

In the prior art, due to the influence of noise, errors and the like,false factors, that is, false obstacles, exist in the obstacles detectedin an obstacle detection process, in order to delete the falseobstacles, in the solution provided by the present application, thefirst view collected by the camera at the moment t and the second viewby the camera at the moment t-1 are obtained. Then, the locationinformation of the same obstacle to be detected in the first view andthe second view is determined, and the motion information of the camerafrom moment t-1 to the moment t is also determined. Then, whether thelocation information of the obstacle to be detected in the two viewsbeing matched with the motion information of the camera are judged, soas to determine whether the obstacle to be detected is the false alarmobstacle. And, if the location information of the obstacle to bedetected in the two views does not match the motion information of thecamera, the obstacle to be detected is judged as the false alarmobstacle. Therefore the false alarm obstacle may be eliminated, and theaccuracy of obstacle detection is ensured.

The embodiment of the present application provides a method for obstacledetection, which mainly calculates location information of obstacleswith different depth thresholds by dividing a disparity map. It shouldbe noted that the location information of the obstacle to be detected inthe first view and the second view in the foregoing embodiment may beobtained based on the method provided by the solution.

The basic principle of the method for detection of false alarm obstacleprovided by the solution is as follows: collecting respectively left andright views of the same scene by using a binocular camera; and thenperforming stereo matching on the collected left and right views toobtain a disparity map in the scene; performing filtering and otherimage preprocessing operations on the disparity map to obtain a finaldisparity map by removing the noise; then dividing the obtaineddisparity map into one or more sub-disparity maps according to differentsub-disparity values range of a disparity value range of the disparitymap; then performing respectively contour detection on the obtainedsub-disparity maps to obtain the contour information of obstaclecontours in the corresponding sub-disparity maps, and finally,calculating the size, the distance, location and other information ofthe obstacle in the scene based on the obtained contour information,internal and external parameters of the camera and depth informationcorresponding to each sub-disparity map.

The target disparity map described below is any one of the firstdisparity map or the second disparity map.

Referring to FIG. 7, the method includes the following steps:

S201. setting a disparity value beyond the sub-disparity value range inthe target disparity map as a first disparity threshold, so as togenerate a sub-disparity map of the target disparity map, based on asub-disparity value range divided by the disparity value range of atarget disparity map.

In the embodiment of the present application, the disparity value in thesub-disparity map beyond the corresponding sub-disparity value range isthe first disparity threshold, that is, the disparity value of a pixelpoint within the sub-disparity value range mainly represented in thesub-disparity map. Exemplarily, the predetermined disparity thresholdmay be set as 0 or 255. Of course, 0 or 255 is only an example, and anydisparity value beyond the sub-disparity value range may be set as thedisparity threshold, which is not limited herein.

It should be noted that the disparity value of the pixel point in thedisparity map in the present application is the pixel value of the pixelpoint.

In an example, since the disparity value is inversely proportional tothe depth, and in order to divide the stereoscopic scene into differentdistances according to the distance, the entire disparity value range ofthe disparity value may divided into a plurality of sub-disparity valueranges according to a certain predetermined interval threshold, anddisparity maps corresponding to different sub-disparity value ranges maybe further obtained.

Based on this, the division process of the disparity value range beforethe step S201 specifically includes the following steps:

S201 a: obtaining a target disparity value range corresponding to apredetermined depth value range from the disparity value range of thefirst disparity map; wherein the disparity value of the pixel point inthe first disparity map is inversely proportional to the depth value ofthe pixel point.

S201 b: dividing the target disparity value range into one or moresub-disparity value ranges according to a predetermined intervalthreshold.

For example, it is assumed that, the distance range that the binocularcamera may detect is [0.5 m, 10.5 m], and the corresponding disparity ofwhich is [255, 0]. An object with a closer distance will generally bethe focus of attention in the obstacle detection. For example, [0.5 m,4.5 m] may be set as a detection range, that is, the predetermined depthvalue range is [0.5 m, 4.5 m], and the corresponding target disparityvalue range is [255,151]. Firstly, all values less than 161 in thedisparity map are set as 0 to obtain a new disparity map D. Then, thescene within the detection range is evenly divided according to thedistance at an interval of lm, the corresponding sub-depth value rangesare respectively [0.5 m, 1.5 m], [1.5 m, 2.5 m], [2.5 m, 3.5 m] and [3.5m, 4.5 m], the predetermined interval threshold of the correspondingdisparity value is 26, and the four corresponding sub-disparity valueranges are respectively [255, 229], [229, 203], [203, 177], and [177,151].

Based on the step SA1 and the step SA2, if the target disparity valuerange in the above step SA2 is composed of one or more sub-disparityvalue ranges which do not overlap, that is, if the entire targetdisparity value range is divided into a plurality of non-overlappingsub-disparity value ranges in the step A2, the sub-disparity mapcorresponding to each sub-disparity value range of the first disparitymap may be obtained according to the step 102.

For example, it is assumed that the target disparity value range of acertain disparity value is [255, 151], the corresponding depth valuerange is [0.5 m, 4.5 m], which is divided into [0.5 m, 1.5 m], [1.5 m,2.5 m], [2.5 m, 3.5 m] and [3.5 m, 4.5 m] at an interval of lm.Thecorresponding sub-disparity maps are the four sub-disparity valueranges, that is,[255, 229], [229, 203], [203, 177] and [177, 151]. Thesub-disparity map corresponding to each sub-disparity value range isobtained respectively. For example, all values less than 229 in thedisparity map D may be set as 0 at first, to obtain a firstsub-disparity map D1; then, values less than 203 in the disparity map Dis set as 0 to obtain a disparity map D1′, and D1′ is subtracted fromthe D1 to obtain a second sub-disparity map D2; then the values lessthan 177 in the disparity map D is set as 0 to obtain a disparity mapD2′, and the D2′ is subtracted from the D2 to obtain a thirdsub-disparity map D3; and finally, the disparity map D is subtractedfrom the disparity map D2′ to obtain a fourth sub-disparity map D4. Forexample, referring to the disparity map shown in FIG. 6, thesub-disparity maps of the corresponding four depth levels arerespectively the first sub-disparity map shown in FIG. 8A, the secondsub-disparity map shown in FIG. 8B, the third sub-disparity map shown inFIG. 8C and the fourth sub-disparity map shown in FIG. 8D.

S202. performing contour detection on the sub-disparity map to obtaincontour information of the obstacle in the sub-disparity map.

In an example, if the sub-disparity maps (e.g., D1, D2, D3, and D4described above) at different depth levels of the disparity map in thesame scene are obtained according to the above example, contourdetection may be performed on each sub-disparity map to obtain thecontour information in each sub-disparity map, that is, the contourinformation of the obstacle. Exemplarily, an information format returnedby each detected contour in each sub-disparity map may be defined as:{“Contour”: x, y, width, height}, wherein x, y represent pixel locationcoordinates in the top left corner (it is only an example herein, and atop right corner, a bottom right corner, a bottom left corner and otherlocations may also be obtained) of the contour and respectivelycorrespond to the located column and row; and width and heightrespectively represent the width and height of the contour, and units ofthe values of which are both pixels.

S203. determining the location of the obstacle according to the contourinformation of the obstacle in the sub-disparity map and the disparityvalue of each pixel point in the sub-disparity map.

For example, referring to FIG. 9, if the sub-disparity maps shown inFIGS. 8A, 8B, 8C, and 8D are taken as an example, the contour detectionis performed on the four images respectively to obtain obstacleinformation corresponding to each depth, and the specific obstaclelocation is shown in FIG. 9.

In an example, when the location information of the obstacle in acertain sub-disparity map is calculated specifically, some location,size and distance information of the obstacle in the scene may be easilycalculated based on the coordinate and size information of differentcontours obtained in the step S103, the parameters of a binocularsystem, and a dense disparity map D obtained above.

For example, taking a certain contour C as an example, the coordinatesof the top left corner thereof are

${\left( {x_{0},y_{0}} \right)\left( {{x_{0} + \frac{w_{0}}{2}},{y_{0} + \frac{h_{0}}{2}}} \right)},$

the width thereof is w0, and the height thereof is h0. Furthermore,images coordinates (x₀+w₀, y₀) and

$\left( {{x_{0} + \frac{w_{0}}{2}},{y_{0} + \frac{h_{0}}{2}}} \right)$

of a top right corner and the center of the contour may be obtained.Firstly, according to the Formula 6:

${{z_{c} = \frac{B \times f}{Disparity}};},$

the disparity value of each point is derived from the disparity mapaccording to the image coordinates, and then the distances Z0, Z1, andZ′ between the scenes corresponding to the top left corner, the topright corner and the central location of the contour and binocularcamera may be obtained. According to the Formula 7:[u,v,1]^(T)=P[X,Y,Z,1]^(T) (a projection relationship between an objectpoint and an image point), the coordinates (X₀,Y₀,Z₀) and (X₁,Y₁,Z₁) ofthe top left corner and the top right corner of the contour in a worldcoordinate system may be respectively calculated, that is, thecoordinates of actual object points corresponding to the top left cornerand the top right corner of the contour. So far, object information inthe contour are all obtained and are defined as {“Obstacle”: X_(L),X_(R), Width, Height, Distance}, wherein X_(L) represents a lateraldistance of the leftmost side from the obstacle to the camera, andX_(L)=X₀; X_(R) represents the lateral distance from the rightmost sideof the obstacle to the camera, and X_(R)=X₁; Width represents the widthof the obstacle, Width=X₁−X₀; Height represents the height of theobstacle, Height=Y₀=Y₁; and Distance represents the axial distance fromthe obstacle to the left and right cameras, Distance=Z′. Furthermore,all obstacle information required for obstacle avoidance is obtained.

Since the disparity value of the pixel point in the disparity map isinversely proportional to the depth of the pixel point, thesub-disparity maps of different depth ranges of the disparity mapcorresponding to the left and right views in the same scene collected bythe binocular camera may be obtained according to the disparity value ofeach pixel point in the disparity map corresponding to the left andright views collected by the binocular camera, and contour detection isperformed on the sub-disparity maps of different depth levels todetermine the locations of obstacles of different depth levels based onthe contour information of the contours at different depth levels,thereby a relative distance between the obstacle and the binocularcamera is accurately positioned. In the solution, simple division isperformed on the disparity map to perform contour detection onsub-disparity maps of different depths, thereby not only shortening thedetection time, but also improving the detection precision.

It should be noted that, before a controlling of the binocular camera tocollect the left view and the right view, it is usually necessary toperform certain adjustment (for example, camera adjustment operationssuch as offline calibration and image correction) for the binocularcamera in advance to ensure that the optical axis of the left camera isparallel to that of the right camera. Then, a baseline length betweenthe optical axis of the left camera and the optical axis of the rightcamera is measured, the focal length of the binocular camera isrecorded, and the baseline length and focal length are ensured to beunchanged, thereby ensuring the synchronization of the images acquiredby the binocular camera and avoiding unnecessary errors.

1) Camera Calibration

The camera calibration in the present application generally refers tooffline calibration for camera. Generally, since the optical axis of thebinocular camera is located inside the camera, it is difficult to ensurethat the optical axes are strictly parallel when the camera isassembled. Generally, there is a certain deviation. Therefore, offlinecalibration may be generally performed on the successfully constructedbinocular camera to obtain the internal parameters (focal length,baseline length, image center, distortion parameters and the like) andexternal parameters (rotation matrix R and translation matrix T) of thecamera.

In one example, the offline calibration is performed on the lens of thebinocular camera by using the Zhang Zhengyou checkerboard calibrationmethod.

When the offline calibration is performed on the camera, the left cameracan be calibrated firstly to obtain the internal and external parametersof the left camera; secondly, the right camera is calibrated to obtainthe internal and external parameters of the right camera; and finally,the binocular camera is calibrated to obtain a rotational translationrelationship between the left and right cameras.

Assuming that there is a point W=[X,Y,Z]^(T) in the world coordinatesystem, the corresponding point of the point on an image plane ism=[u,v]^(T), then the projection relationship between the object pointand the image point is:

[u,v,1]^(T)=P[X,Y,Z,1]^(T)   (Formula 10);

Wherein P represents a 3×4 projection matrix, which may be representedby a rotation and translation matrix:

P=A[Rt]  (Formula 11);

Wherein R represents a 3×3 rotation matrix and t represents atranslation vector, the two matrices represent the external parametersof binocular vision, wherein one represents location and the otherrepresents direction. In this way, the location of each pixel point onthe image in the world coordinate system may be determined. Wherein amatrix represents an internal parameter matrix of the camera and may beexpressed as follows:

$A = {\begin{pmatrix}f_{u} & \beta & u_{0} \\0 & f_{v} & v_{0} \\0 & 0 & 1\end{pmatrix}.}$

In the above formula, (u_(o), v_(o)) represents the coordinate of thecenter point of the image; f_(u) and f_(v) respectively represent thefocal length represented by horizontal and vertical pixel units, and βrepresents a tilt factor.

Some parameters obtained in the above offline calibration process areapplied to both image correction and obstacle calculation processes.

2) Image Correction

Since the lens distortion causes distortion to an image collected by thelens, distortion correction and polar line correction may be generallyperformed on the binocular camera before the binocular camera collectsimages. Assuming that an undistorted reference image is f(x, y) and theimage with larger geometric distortion is g(x′, y′), the set distortionbetween coordinate systems for the two images may be expressed as:

$\left\{ {\begin{matrix}{x^{\prime} = {h_{1}\left( {x,y} \right)}} \\{y^{\prime} = {h_{2}\left( {x,y} \right)}}\end{matrix};} \right.$

and the above formula is expressed by a binary polynomial:

$\begin{matrix}{{x^{\prime} = {\sum\limits_{i = 0}^{n}\; {\sum\limits_{j = 0}^{n = i}\; {a_{ij}x^{i}y^{i}}}}}{{y^{\prime} = {\sum\limits_{i = 0}^{n}\; {\sum\limits_{j = 0}^{n = i}\; {b_{ij}x^{i}y^{i}}}}};}} & \left( {{Formula}\mspace{14mu} 12} \right)\end{matrix}$

Wherein n represents the coefficient of the polynomial, i and jrepresent the specific locations of the pixel points in the image, anda_(ij) and b_(ij) represent various coefficients. An image subject todistortion correction is obtained by the above formula.

A polar line correction operation of the image is based on the rotationand translation matrices of the left and right cameras obtained in theoffline camera correction. It is assumed that the rotation andtranslation matrices of the left camera are R1 and t1, and the rotationand translation matrices of the right camera are R2 and t2, and therotation and translation matrices may be obtained during the offlinecorrection. Based on the rotation and translation matrices of the leftand right cameras, polar lines corresponding to the images of the leftand right cameras are parallel to each other by using a Bouguet polarline correction method. The time complexity of stereo matching isgreatly reduced, and the disparity calculation process is simplified.

The solution provided by the embodiment of the present application ismainly introduced from the perspective of the apparatus for detection offalse alarm obstacle and the terminal used by the apparatus. It may beunderstood that the apparatus includes corresponding hardware structuresand/or software modules for performing various functions, in order toimplement the above functions. Those skilled in the art will readilyappreciate that the present application may be implemented by hardwareor a combination of hardware and computer software, in combination withthe units and algorithm steps of the various examples described in theembodiments disclosed herein. Whether a certain function is implementedin the form of hardware or a driven hardware by the computer software isdetermined by specific applications and designed constraint conditionsof the technical solutions. Those skilled in the art may implement thedescribed functions by using different methods for each specificapplication, but this implementation should not be considered beyond thescope of the present application.

In the embodiment of the present application, the function modules ofthe apparatus for detection of false alarm obstacle may be dividedaccording to the above method example. For example, the function modulesmay be divided according to the corresponding functions, and two or morefunctions may also be integrated into one processing module. The aboveintegrated module may be implemented in the form of hardware and mayalso be implemented in the form of a software function module. It shouldbe noted that the division of the modules in the embodiment of thepresent application is schematic and is only a logical functiondivision, and other division manners may be provided during the actualimplementation.

The apparatus embodiment provided by the embodiment of the presentapplication, corresponding to the method embodiment provided above, isdescribed below. It should be noted that the explanation of relatedcontents in the following apparatus embodiment may refer to theforegoing method embodiment.

In the case that the function modules are divided according to thecorresponding functions, FIG. 10 shows a possible structural schematicdiagram of the apparatus for detection of false alarm obstacle involvedin the above embodiment. The apparatus 4 includes: an obtaining module31, a determining module 32 and a judging module 33. The obtainingmodule 31 is configured to support the apparatus for detection of falsealarm obstacle to execute the process S101 in FIG. 2; the determiningmodule 32 is configured to support the apparatus for detection of falsealarm obstacle to execute the processes S102 and S103 in FIG. 2; and thejudging module 33 is configured to support the apparatus for detectionof false alarm obstacle to execute the process S104 in FIG. 2. Further,the judging module 33 is specifically configured to support theapparatus to execute the processes SA1, S104 a-S104 a 3 in FIG. 3.Further, the judging module 33 is specifically configured to support theapparatus to execute the processes S104 b 1-S104 b 3 in FIG. 4. Further,the determining module 32 is further configured to support the apparatusto execute the steps S201-S203 in FIG. 7. Further, the determiningmodule 32 is further configured to support the apparatus to execute thestep S103 a in the above context. All the related contents of the stepsinvolved in the foregoing method embodiment may be quoted to thefunction descriptions of the corresponding function modules, and thusdetails are not described herein again.

In hardware implementation, the obtaining module 31, the determiningmodule 32 and the determining module 33 may be processors. The programscorresponding to the actions executed by the apparatus for detection offalse alarm obstacle described above may be stored in a memory of theapparatus in the form of software, so that the processor loads andexecute the operations corresponding to the above respective modules.

FIG. 11 shows a possible structural schematic diagram of an electronicdevice involved in the embodiment of the present application. Theapparatus 4 includes a processor 41, a memory 42, a system bus 43 and acommunication interface 44. The memory 42 is configured to storecomputer execution codes; the processor 41 is connected with the memory42 through the system bus 43. When the apparatus is running, theprocessor 41 is configured to execute the computer execution codesstored in the memory 42 to execute any method for detection of falsealarm obstacle provided by the embodiment of the present application,for example, the processor 41 is configured to support the apparatus toexecute all steps in FIG. 3, and/or applied to other processes of thetechnology described herein, the specific false alarm obstacle detectionmethod may be referred to related descriptions below and in thedrawings, and thus are not described herein again.

The embodiment of the present application further provides a storagemedium, which may include a memory 42.

The embodiment of the present application further provides a computerprogram, the computer program may be directly loaded into the memory 42and contains software codes, and the computer program may implement theabove method for detection of false alarm obstacle after being loadedand executed by a computer.

The processor 41 may be a single processor and may also be a unit nameof a plurality of processing elements. For example, the processor 41 maybe a central processing unit (central processing unit, CPU). Theprocessor 41 may also be other general purpose processors, a digitalsignal processor (digital signal processor, DSP), an applicationspecific integrated circuit (application specific integrated circuit,ASIC), a field-programmable gate array (field-programmable gate array,FPGA) or other programmable logic devices, discrete gates or transistorlogic devices, discrete hardware components and the like, and theprocessor may implement or execute logic boxes, modules and circuits ofvarious examples described in combination with the contents disclosed bythe present application. The general purpose processor may be amicroprocessor or the processor may also be any conventional processoror the like. The processor 41 may also be a dedicated processor, and thededicated processor may include at least one of a baseband processingchip, a radio frequency processing chip and the like. The processor mayalso be a combination for implementing a computing function, forexample, a combination including one or more microprocessors, acombination of a DSP and a microprocessor, and the like. Further, thededicated processor may further include a chip having other dedicatedprocessing functions of the apparatus.

The steps of the method described in combination with the contentsdisclosed by the present application may be implemented in the form ofhardware and may also be implemented by a processor executing softwareinstructions. The software instruction may be composed of correspondingsoftware modules, and the software modules may be stored in a randomaccess memory (English: random access memory, abbreviation: RAM), aflash memory, a read only memory (English: read only memory,abbreviation: ROM), an erasable programmable read-only memory (English:erasable programmable ROM, abbreviation: EPROM), an electricallyerasable programmable read-only memory (English: electrical EPROM,abbreviation: EEPROM), a register, a hard disk, a mobile hard disk, aCD-ROM (CD-ROM) or any other form of storage medium known in the art. Anexemplary storage medium is coupled to the processor, so that theprocessor may read information from and write information to the storagemedium. Of course, the storage medium may also be a constituent part ofthe processor. The processor and the storage medium may be located in anASIC. Additionally, the ASIC may be located in a terminal device. Ofcourse, the processor and the storage medium may also exist as discretecomponents in the terminal device.

The system bus 43 may include a data bus, a power bus, a control bus anda signal state bus and the like. For the sake of clarity in the presentembodiment, various buses are illustrated as the system bus 43 in FIG.11.

The communication interface 44 may specifically be a transceiver on theapparatus. The transceiver may be a wireless transceiver. For example,the wireless transceiver may be an antenna or the like of the apparatus.The processor 41 communicates with other devices via the communicationinterface 44. For example, if the apparatus is a module or component inthe terminal device, the apparatus is applied to the data interactionwith other modules in the electronic device.

The embodiment of the present application further provides a robot,including the apparatus for detection of false alarm obstaclecorresponding FIG. 10 and FIG. 11.

Those skilled in the art should be aware that, in one or more examplesdescribed above, the functions described in the present application maybe implemented by hardware, software, firmware, or any combinationthereof. When implemented by the software, these functions may be storedin a computer readable medium or transmitted as one or more instructionsor codes on the computer readable medium. The computer readable mediumincludes a computer storage medium and a communication medium, whereinthe communication medium includes any medium that may convenientlytransfer the computer programs from one place to another. The storagemedium may be any available medium that may be accessed by a generalpurpose or special purpose computer.

Finally, it should be noted that, the objects, technical solutions andbeneficial effects of the present application have been furtherillustrated in detail by the above specific embodiments. It should beunderstood that the foregoing descriptions are merely specificembodiments of the present application, but the protection scope of thepresent application is not limited thereto. Any modifications,equivalent substitutions and improvement and the like made on the basisof the technical solutions of the present application shall fall withinthe protection scope of the present application.

1. A method for detection of false alarm obstacle, comprising: obtaininga first view at a moment t and a second view at a moment t-1, collectedby a camera; determining location information of the same obstacle to bedetected in the first view and the second view; determining motioninformation of the camera from moment t-1 to the moment t; judgingwhether the location information of the same obstacle to be detected intwo views being matched with the motion information of the camera; andif not, judging the obstacle to be detected as a false alarm obstacle.2. The method according to claim 1, wherein, said judging whether thelocation information of the same obstacle to be detected in two viewsbeing matched with the motion information of the camera comprises:determining the motion information of the obstacle to be detectedaccording to the location information of the obstacle to be detected inthe two views; judging whether the motion information of the obstacle tobe detected is consistent with the motion information of the camera; andwhen the motion information of the obstacle to be detected isinconsistent with the motion information of the camera, judging that thelocation information of the same obstacle to be detected in the twoviews being not matched with the motion information of the camera. 3.The method according to claim 1, wherein said judging whether thelocation information of the same obstacle to be detected in two viewsbeing matched with the motion information of the camera comprises:estimating estimated location information of the obstacle to be detectedin the first view, according to the motion information of the camera andthe location information of the obstacle to be detected in the secondview; judging whether the estimated location information of the obstacleto be detected in the first view is consistent with the locationinformation of the obstacle to be detected in the first view; and if theestimated location information of the obstacle to be detected in thefirst view is inconsistent with the location information of the obstacleto be detected in the first view, judging that the location informationof the obstacle to be detected in the two views being not matched withthe motion information of the camera.
 4. The method according to claim2, wherein the camera is a binocular camera; the first view is adisparity map from the left view and the right view collected by thebinocular camera at the moment t; and the second view is a disparity mapfrom the left view and the right view collected by the binocular cameraat the moment t-1; and said determining motion information of the camerafrom moment t-1 to the moment t comprises: determining the motioninformation of the camera according to the location information of otherobjects except for the obstacle to be detected in the first view and thelocation information of the other objects in the second view.
 5. Themethod according to claim 4, wherein a target view comprises the firstview or the second view, and determining the location information of theobject in the target view comprises: based on a sub-disparity valuerange divided from a disparity value range of the target view, setting adisparity value beyond the sub-disparity value range in the target viewas a first disparity threshold, so as to generate a sub-view of thetarget view; performing contour detection on the sub-view to obtaincontour information of the object in the sub-view; and determining thelocation information of the object according to the contour informationof the object and the disparity value of each pixel point in thesub-disparity map.
 6. An electronic device, wherein the electronicdevice comprises a memory and a processor, the memory is coupled in theprocessor and is configured to store computer software codes, and thecomputer software codes are configured to control the processor toexecute the method according to claim
 1. 7. A computer storage medium,for storing computer software instructions used by the apparatus fordetection of false alarm obstacle and containing program codes designedto execute the method according to claim 1.